Since the first commercially successful use of a GUI in the 1980s, the GUI has been adopted by most developers of computer operating systems and application software, and it is now used on or available for virtually all types of computers; GUIs are also employed, in differing forms, in a wide range of other products, such as industrial controls, televisions and other monitors, electronic games and toys. Typically, a user interacts with a GUI using a mouse, track ball, touch pad or the like, often in concert with a keyboard; interaction is also possible with a keyboard alone, however users find it faster, easier and often more intuitive to use a mouse, etc. rather than just a keyboard. The use of such device combines with the advantageously intuitive characteristics of the GUI to provide users with a simple way to interact with computers and the like, such that the use of such devices as mice, touchpads, trackballs, joysticks, etc. to provide a primary means of interaction with a GUI is almost ubiquitous today (with a keyboard, console or other control providing an important, secondary means of interaction).
The rapid and continuing development of electronic devices and software, and the massive increases in processing power and memory now available to the general population have begun to highlight limitations and disadvantages of typical GUI interface or interaction devices such as mice, touchpads, track balls and the like. Mice need to be positively held by a user, and need a preferably flat surface upon which they can be moved in order to effect many of their functions. Because they are designed to be moved over a flat surface, mice can only effectively provide control over a two-dimensional GUI (which limits their use in applications such as computer aided design or games, which are very often three-dimensional, or with the emerging three-dimensional displays and TVs); moreover, the range of control movement offered by a mouse is restricted by the size of the surface upon which the mouse sits (which is one reason why mice often incorporate a “scroll wheel”). Such limitations and disadvantages apply, to a greater or lesser extent, to similar GUI interface devices such as touchpads, trackballs and the like.
In computer gaming, manufacturers have attempted to address interaction with the GUI (or game), in different ways: initially, control/interaction was carried out through specialised consoles, which incorporated many of the functions of mice, etc., but in a different format. These consoles began to be equipped with accelerometers, or other means which were able to recognise motion of the console (such as by the console having an infra-red transmitter, with movement of the console being monitored by remote infra-red sensors) and to translate this recognised movement into instructions for player-game interaction. Consoles designed to be held in two hands were replaced by smaller consoles which a user could hold in a single hand, allowing a user to play electronic games such as tennis or golf. A later and ongoing advance in the gaming field is to provide a way of registering movement or gestures made by the user (as opposed to something held or carried by the user) and translating these into instructions for the game; such an approach is now also being adopted by the manufacturers of other equipment, such as the so-called “smart” TVs, where it is becoming more common to enable a user to interact with the GUI by way of gestures alone. Such an approach is attractive, because it facilitates a very intuitive interaction, however it has a number of disadvantages. It requires sophisticated and sensitive sensors in order first to capture the movement of a user, then it requires significant amounts of processing capability to analyse the movement and to translate it into the instructions or controls which are intended by the user and also causing significant latency; as a result, such systems today are reasonably reliable in interpreting gross movements and converting these into controls to the GUI, but less so where smaller or more precise and/or rapid movements or gestures are concerned. They often have a limited field of view, limiting the range of movement of the user, or require some way of tracking the user, they risk being confused by movements other than by those of the user, and because they are most reliable with gross movements they are inherently slow to respond and limited in the range of different functionalities they can currently provide, because the range of gross movements available to a user is limited.